Different results when using 'foreach of local' and 'foreach in `local-name''

[ZHIDI ZHANG]

Dear statalist:

I would like to compose nested loops to get regression results. To my knowledge, the function of 'foreach of local' and 'foreach in `local-name'' are the same. But when I run the dotfiles, it seems like 4 fewer variables in the `of' than in the `in' format.

To keep things simple, my example shows what it would look like as following code. The results ：
The first 'of local jlist' last displayed m = 4, while the 'in `jlist'' lasted displayed m = 7

Thanks a lot for any kind help!

global Mediator "lnexercise_c lnExpense_healthcare_c lnExpense_education_c lnExpense_bussCare_c lnwateruse_c lnfireuse_c lnelectricuse_c"

local jlist $Mediator

local m = 0
foreach j of local jlist {
 gettoken j jlist : jlist
 sum `j'
 
 local `++m'
 display `m'
}

foreach j in `jlist' {
 gettoken j jlist : jlist
 sum `j'
 
 local `++m'
 display `m'
}

[ZHIDI ZHANG]

The dataset example looks like:

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(lnexercise_c lnExpense_healthcare_c lnExpense_education_c lnExpense_bussCare_c lnwateruse_c lnfireuse_c lnelectricuse_c)
1.7917595         0  7.78364 0 1.0986123  .6931472 1.3862944
        0  7.601402        0 0 1.0986123 1.3862944 1.0986123
1.3862944  7.313887        0 0 1.0986123  .6931472 1.3862944
        0  8.575651 6.908755 0 1.0986123 1.3862944 1.3862944
        0 3.9318256 8.853808 0 1.0986123  .6931472 1.3862944
        0         0 8.517393 0 1.0986123  .6931472  1.609438
        0  5.993961 5.916202 0 1.0986123 1.3862944  1.609438
        0 3.9318256 6.552508 0 1.0986123 1.3862944  1.609438
        0  5.303305 6.552508 0 1.0986123 1.3862944  1.609438
1.0986123  7.313887        0 0 1.0986123  .6931472 1.3862944
end

[Clyde Schechter]

To my knowledge, the function of 'foreach of local' and 'foreach in `local-name'' are the same.

Your knowledge is not quite correct.

With -foreach m in `jlist'-, `jlist' is expanded once at the top of the loop, and only then. If the code inside the loop changes the contents of jlist, nevertheless the iterator m goes through all and only the original values.

By contrast, with -foreach m of local jlist-, local macro jlist is re-evaluated at each iteration of the loop. So if the code inside the loop modifies the contents of jlist, the loop will, at each iteration, use the updated contents of jlist, not the original contents.

Your loop does modify the contents of jlist. At each iteration, it pulls the first remaining item off of jlist, leaving it one item shorter. Your second loop then starts off with the a jlist that only contains the final three elements that were not removed during the first loop. If you re-initialize -local jlist $Moderator before the second loop you will see that part of the difference between the two loops is removed. But, even with that, a difference remains just because different elements are being iterated over in the two loops.

Finally, there is a third complication in your code: you are also modifying the value of the iterator `j' inside the loop. That leads to undefined behavior and should never be done.

Added: Crossed with #2.
Added (after #4, though not in response to it). This is explained in the PDF documentation on -foreach-:

If the contents of flist are modified in the body of foreach file in `flist', foreach will
not notice, and the original list will be used. The contents of flist may, however, be modified in
foreach file of local flist, but only to add new elements onto the end.

Note the additional point that it is not licit to modify the contents of jlist in -foreach j of local jlist- by chopping things out of it as was done in your code. Only adding things onto the end is licit. I should point out, however, that even the latter can be ill-advised. For example:

local mylist abc
foreach m of local mylist {
    display "`m'"
    local mylist `mylist' `m'
}

is, according to the manual, allowed. But it results in an infinite loop. So even licit modification of the local macro should be done only with great caution.

Added yet again: Crossed with #5.

[Nick Cox]

The gettoken commands are better omitted here, as at best redundant and at worst too much like sawing off the tree branch you're sitting on.

More prosaically, repeated use of gettoken is sometimes a good way to loop through a list, but if it's used to modify the macro you're working on, that can be somewhere between puzzling and worse.

https://journals.sagepub.com/doi/pdf...6867X211063415 was an attempt at a tutorial survey in this territory.

[Daniel Schaefer]

Notice that you don't reinitialize m to 0 after before the second loop, so you just start counting from where the first loop stops. The first loop iterates 4 times, and the second iterates 3 times, not 7 times. Your getting this strange behavior because you are mutating the local `jlist' each time you call the gettoken command on each iteration of the loop. You can fix these issues like so:

global Mediator "lnexercise_c lnExpense_healthcare_c lnExpense_education_c lnExpense_bussCare_c lnwateruse_c lnfireuse_c lnelectricuse_c"

local jlist $Mediator

local m = 0
foreach j of local jlist {
 gettoken j : jlist
 qui sum `j'
 
 local `++m'
 display `m'
}

local m = 0
foreach j in `jlist' {
 gettoken j : jlist
 qui sum `j'
 local `++m'
 display `m'
}

. local jlist $Mediator

.
. local m = 0

. foreach j of local jlist {
  2.  gettoken j : jlist
  3.  qui sum `j'
  4.  
.  local `++m'
  5.  display `m'
  6. }
1
2
3
4
5
6
7

.
. local m = 0

. foreach j in `jlist' {
  2.  gettoken j : jlist
  3.  qui sum `j'
  4.  local `++m'
  5.  display `m'
  6. }
1
2
3
4
5
6
7

But you probably shouldn't use gettoken at all for this. If you're sure you want to globalize the mediator macro, just loop like this:

global Mediator "lnexercise_c lnExpense_healthcare_c lnExpense_education_c lnExpense_bussCare_c lnwateruse_c lnfireuse_c lnelectricuse_c"

local m = 0
foreach j of varlist $Mediator {
 sum `j'
 
 local `++m'
 display `m'
}

local m = 0
foreach j in $Mediator {
 sum `j'
 
 local `++m'
 display `m'
}

"of" and "in" aren't equivalent. "of" will check the type of the input, and "of varlist" will throw an error if one of the tokens is not a variable. In will not. Also, a quick note on terminology: these for loops are not "nested." for loops are "nested" if one appears within another, like this:

forv i = 1/10{
    forv j = 1/10{
        di "`i', `j'"
    }
}

Edit: Crossed with #3 and #4. See also #6.

[Daniel Schaefer]

Actually, what I have in #3 is not entirely correct. My -gettoken- loop iterates the correct number of times, but it doesn't implement -gettoken- correctly. It looks like maybe -gettoken- is intended to mutate the local as a way of tracking the current state. If you are going to use it, then you shouldn't iterate over `jlist' (because it's being mutated) and although you can get away with mutating j on each iteration, it is probably bad practice. I think this is how one should actually implement -gettoken-:

global Mediator "lnexercise_c lnExpense_healthcare_c lnExpense_education_c lnExpense_bussCare_c lnwateruse_c lnfireuse_c lnelectricuse_c"

local jlist $Mediator

local m = 0
local words = wordcount("`jlist'")
forv i = 1/`words' {
    gettoken j jlist : jlist
    sum `j'

    local `++m'
    display `m'
}

But again, this is a pretty idiosyncratic approach, and you probably have better options than to use -gettoken- in the first place.

[ZHIDI ZHANG]

The gettoken commands are better omitted here, as at best redundant and at worst too much like sawing off the tree branch you're sitting on.

More prosaically, repeated use of gettoken is sometimes a good way to loop through a list, but if it's used to modify the macro you're working on, that can be somewhere between puzzling and worse.

https://journals.sagepub.com/doi/pdf...6867X211063415 was an attempt at a tutorial survey in this territory.

I'll look into the details of it~ Thank you so much!

[ZHIDI ZHANG]

Actually, what I have in #3 is not entirely correct. My -gettoken- loop iterates the correct number of times, but it doesn't implement -gettoken- correctly. It looks like maybe -gettoken- is intended to mutate the local as a way of tracking the current state. If you are going to use it, then you shouldn't iterate over `jlist' (because it's being mutated) and although you can get away with mutating j on each iteration, it is probably bad practice. I think this is how one should actually implement -gettoken-:

global Mediator "lnexercise_c lnExpense_healthcare_c lnExpense_education_c lnExpense_bussCare_c lnwateruse_c lnfireuse_c lnelectricuse_c"

local jlist $Mediator

local m = 0
local words = wordcount("`jlist'")
forv i = 1/`words' {
gettoken j jlist : jlist
sum `j'

local `++m'
display `m'
}

But again, this is a pretty idiosyncratic approach, and you probably have better options than to use -gettoken- in the first place.

Many Thanks! I'm grateful to you.

[ZHIDI ZHANG]

Your knowledge is not quite correct.

With -foreach m in `jlist'-, `jlist' is expanded once at the top of the loop, and only then. If the code inside the loop changes the contents of jlist, nevertheless the iterator m goes through all and only the original values.

By contrast, with -foreach m of local jlist-, local macro jlist is re-evaluated at each iteration of the loop. So if the code inside the loop modifies the contents of jlist, the loop will, at each iteration, use the updated contents of jlist, not the original contents.

Your loop does modify the contents of jlist. At each iteration, it pulls the first remaining item off of jlist, leaving it one item shorter. Your second loop then starts off with the a jlist that only contains the final three elements that were not removed during the first loop. If you re-initialize -local jlist $Moderator before the second loop you will see that part of the difference between the two loops is removed. But, even with that, a difference remains just because different elements are being iterated over in the two loops.

Finally, there is a third complication in your code: you are also modifying the value of the iterator `j' inside the loop. That leads to undefined behavior and should never be done.

Added: Crossed with #2.
Added (after #4, though not in response to it). This is explained in the PDF documentation on -foreach-:

Note the additional point that it is not licit to modify the contents of jlist in -foreach j of local jlist- by chopping things out of it as was done in your code. Only adding things onto the end is licit. I should point out, however, that even the latter can be ill-advised. For example:

local mylist abc
foreach m of local mylist {
display "`m'"
local mylist `mylist' `m'
}

is, according to the manual, allowed. But it results in an infinite loop. So even licit modification of the local macro should be done only with great caution.

Added yet again: Crossed with #5.

Thank you so much! I appreciate your patience, insight, and quick response!
I learned a lot from this reply.